Determining Emotions Using Camera-Based Sensing

ABSTRACT

In one embodiment, a computer-readable non-transitory storage medium embodies software that is operable when executed to, in real time, capture, by a single sensor, a number of images of a user; determine, based on the number of images, one or more short-term cardiological signals of the user during a period of time; estimate, based on the cardiological signals, a first short-term emotional state of the user; determine, based on the number of images, one or more short-term neurological signals of the user during the period of time; estimate, based on the neurological signals, a second short-term emotional state of the user; compare the first estimated emotional state to the second estimated emotional state; and in response to a determination that the first estimated emotional state corresponds to the second estimated emotional state, determine a short-term emotion of the user during the period of time.

PRIORITY

This application claims the benefit, under 35 U.S.C. § 119(e), of U.S.Provisional Patent Application No. 62/500,277 filed 2 May 2017, U.S.Provisional Patent Application No. 62/492,838 filed 1 May 2017, and U.S.Provisional Patent Application No. 62/510,579 filed 24 May 2017, all ofwhich are incorporated herein by reference.

TECHNICAL FIELD

This disclosure generally relates to electronic detection and evaluationof an individual's physiological characteristics.

BACKGROUND

An increasing interest in the evaluation human emotions and/orphysiological characteristics, such as cardiovascular health, may beseen in the behavioral, biological, and social sciences. For example,many phenomena, ranging from individual cognitive processing to socialand collective behavior, may not be well understood without taking humanemotions into account. Recently, there has been an interest in affectivecomputing, which is the study and development of computing systems ordevices that can recognize, interpret, process, and simulate humanaffects. In particular, the emotion of the user may be used as an inputfor subsequent operations by computing devices or systems. In addition,determining physiological characteristics is useful for detecting orpredicting the health of the user.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example network environment.

FIG. 2 illustrates example optical measurement areas on the face andneck of a user.

FIG. 3 illustrates an example calculation of cardiological metrics.

FIG. 4 illustrates an example method for calculating cardiologicalmetrics for a user.

FIG. 5 illustrates an example method for performing heartratemeasurements for a user.

FIG. 6A illustrates an example of a baseline data histogram.

FIG. 6B illustrates an example of a test data histogram in which asympathovagal balance is shifted to the left.

FIG. 6C illustrates an example of a test data histogram in which asympathovagal balance is shifted to the right.

FIG. 7 illustrates an example diagram of the circumplex model ofemotion.

FIG. 8 illustrates an example method for emotion evaluation.

FIG. 9 illustrates an example computer system according to particularembodiments of the invention.

DESCRIPTION OF EXAMPLE EMBODIMENTS

FIG. 1 illustrates an example network environment 100. Networkenvironment 100 includes one or more optical cameras 110, a clientsystem 120, a network 130, one or more servers 140, and one or more datastores 150. Optical camera 110, client system 120 servers 140, and datastores 150 may be connected to each other by network 130 via links 160.In particular embodiments, optical camera 110 may be integrated withclient system 120. Although FIG. 1 illustrates a particular arrangementof optical camera 110, client system 120, network 130, servers 140, anddata stores 150, this disclosure contemplates any suitable arrangementof user 110, client system 120, network 130, servers 140, and datastores 150.

Network environment 100 includes one or more optical camera 110 that maybe connected to client system 120. In particular embodiments, opticalcamera 110 is configured to measure data from one or more systems of ahuman body. As an example and not by way of limitation, optical camera110 may be configured to monitor cardiological activity of the humanbody, as described below. As another example, optical camera 110 may beconfigured to monitor neurological and neuroanatomical activity of thehuman body, as described below.

In particular embodiments, optical camera 110 may connect to clientsystem 120 directly or via network 130, which may facilitate interactionbetween and/or transfer of data between optical camera 110 and clientsystem 120. Data (e.g., heart rate, emotional state, etc.) may be storedon client systems 120, data stores 150, other suitable databases, or anycombination thereof. In addition, the processing of the data andcomputations of particular algorithms (as discussed below) may beperformed by client system 120, servers 140, other suitabledevices/systems, or any combination thereof. In particular embodiments,the processing of the data and computations of particular algorithms maybe performed by accessing user data, frame of reference/baseline data,medical data, other relevant data, or any combination thereof, from datastores 150 via network 130.

As an example and not by way of limitation, two or more of clientsystems 120, servers 140, data stores 150, or any suitable combinationthereof may be connected to each other directly (e.g., Ethernet or localarea network (LAN) connection), bypassing network 130. As anotherexample, two or more of client system 120, servers 140, and data stores150 may be physically or logically co-located with each other in wholeor in part. In particular embodiments, client system 120 may be anysuitable computing device, such as, for example, a mobile computingdevice, a smartphone, a cellular telephone, a tablet computer, a laptopcomputer, a personal computer, or any combination thereof. In addition,these devices may communicate with each other via network 130, directly(e.g., by non-network connections), by any other suitable methods, orany combination thereof. As an example and not by way of limitation,client system 120 may communicate with network 130 via a wirelesscommunications protocol, such as Wi-Fi or BLUETOOTH. In particularembodiments, network 130 may be any suitable network. As an example andnot by way of limitation, one or more portions of network 130 mayinclude an ad hoc network, an intranet, an extranet, a virtual privatenetwork (VPN), a local area network (LAN), a wireless LAN (WLAN), a widearea network (WAN), a wireless WAN (WWAN), a metropolitan area network(MAN), a portion of the Internet, a portion of the Public SwitchedTelephone Network (PSTN), a cellular telephone network, or a combinationof two or more of these. Network 130 may include one or more networks.

In particular embodiments, links 160 may connect client system 120,servers 140, and data stores 150 to network 130 or to each other. Thisdisclosure contemplates any suitable links 160. In particularembodiments, one or more links 160 include one or more wireline (such asfor example Digital Subscriber Line (DSL) or Data Over Cable ServiceInterface Specification (DOCSIS)), wireless (such as for example Wi-Fior Worldwide Interoperability for Microwave Access (WiMAX)), or optical(such as for example Synchronous Optical Network (SONET) or SynchronousDigital Hierarchy (SDH)) links. In particular embodiments, one or morelinks 160 each include an ad hoc network, an intranet, an extranet, aVPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, a portion of the Internet, aportion of the PSTN, a cellular technology-based network, a satellitecommunications technology-based network, another link 160, or acombination of two or more such links 160. Links 160 need notnecessarily be the same throughout network environment 100, such thatone or more first links 160 may differ in one or more respects from oneor more second links 160.

In particular embodiments, servers 140 may be any suitable servers. Eachserver 140 may be a unitary server or a distributed server spanningmultiple computers or multiple datacenters. Servers 140 may be ofvarious types, such as, for example and without limitation, web server,file server, application server, exchange server, database server, proxyserver, another server suitable for performing functions or processesdescribed herein, or any combination thereof. In particular embodiments,each server 140 may include hardware, software, or embedded logiccomponents or a combination of two or more such components for carryingout the appropriate functionalities implemented or supported by server140.

In particular embodiments, data stores 150 may be any suitable datastores. Data stores 150 may be used to store various types ofinformation. In particular embodiments, the information stored in datastores 150 may be organized according to specific data structures. Inparticular embodiments, each data store 150 may be a relational,columnar, correlation, or other suitable database. Data store 150 mayinclude networked storage such as cloud storage or other networkaccessible storage. Additionally or alternatively, data store 150 mayinclude local storage within or directly attached to any of the devicesof client system 120, such as solid state drives (SSDs) or hard diskdrives (HDDs).

In particular embodiments, data store 150 may store various datastructures relevant to an optical detection device and the processing ofdata collected by the optical detection device. As an example and not byway of limitation, data store 150 may store a data structurecorresponding to neurocardiology measurements (e.g., measurements ofheart rate (HR), heart-rate variability (HRV), data derived from heartrate (HR) or heart-rate variability (HRV) (e.g., sympathovagal balance(SVB)), neurological data, or neuroanatomical data (e.g.,lateralization, posture, gestures, or context). As another example andnot by way of limitation, data store 150 may store a data structurecorresponding to features data and features vectors determined based ona features evaluation process for emotion evaluation. Although thisdisclosure describes or illustrates particular types of components anduses of these components of network environment 100, this disclosurecontemplates any suitable types of components, any suitable networktopology (e.g., including a standalone-device topology), and anysuitable uses for these components of network environment 100.

Particular embodiments described below describe monitoring andmeasurement of pathophysiological interplays of the nervous andcardiovascular systems through data collected by optical camera 110.Neurocardiology metrics may be measured in the context of mobilemonitoring for use in personalized health evaluation. In mobilemonitoring, use of optical camera 110 may facilitate real-time mobilehealth evaluation. In particular embodiments, neurocardiology metricsmay be indicators for various cardiac ailments or conditions. Herein,reference to real-time refers to measurements performed withinapproximately 5 seconds or less.

The automated evaluation of human emotions is useful in the behavioral,biological, and social applications, among others. Particularembodiments discussed below describe the evaluation of human emotions inthe context of mobile monitoring use in personalized entertainmentexperiences. In mobile monitoring, use of optical camera 110 mayfacilitate real-time evaluation of human emotions. In particularembodiments, human emotions are evaluated by measuring theactivities/arousal of the autonomic nervous system (ANS), where changesin arousal are indicators for various emotions.

FIG. 2 illustrates example optical measurement areas on the face andneck of a user. In particular embodiments, the optical camera maymeasure the blood volume with each pulse (BVP) at a particular region ofinterest (ROI) using photoplethysmogram (PPG). PPG illuminates the skinof a user and determines the underlying cardiovascular activity based onchanges in light absorption. As an example and not by way of limitation,PPG may measure a change in blood volume caused by a pressure pulse byilluminating the skin (e.g., by ambient light or from a light-emittingdiode (LED) of the optical camera) and measuring the amount of reflectedlight at particular frequencies using the image sensor of the opticalcamera. The PPG signal is proportional to the volume of blood in theROI.

In particular embodiments, PPG measurements may be performed with afocus on the face of the user which is relatively simple to track andhas multiple distinctive features (e.g., eyes, mouth, or nose). By usinga known layout of arteries on the human body, the blood volume andstiffness analysis may focus on major but superficial (close to thesurface) arteries. Superficial arteries are accessible using light withparticular wavelengths (e.g., corresponding to the colors red, green,and blue) and a measuring signal modulated by the pulsatile blood inthese larger blood vessels may be captured using the optical camera. Inparticular embodiments, arteries may be selected in advance when thecamera is being setup. Images of particular areas of face 202 arecaptured. As illustrated in the example of FIG. 2, the image of the face202 of the user may be partitioned by bounding boxes 204A-D that eachcorrespond to a particular ROI. The ROIs may correspond to one or morearteries of face 202. In particular embodiments, data may be extractedfrom images of the optical camera defined by bounding boxes 204A-D.

By positioning one or more of the bounding boxes 204A-D along knownorientations corresponding to particular superficial arteries, acomputing system may determine blood volume pulse from fluctuations inthe pulsatile blood. A map of several arteries located at variedlocations of the face may be created based on the signal quality (e.g.,signal strength), such that even when a person is mobile at least one ofthe arteries may still be located. As an example and not by way oflimitation, one or more of the bounding boxes 204A-D may correspond toone or more of the angular, maxillary, ophthalmic, or facial arteries.The superficial arteries may be located using pre-determined humanarterial layout information. In particular embodiments, the location ofsuperficial arteries on a particular user's face may be refined duringthat user's baseline measurements. For example, a device incorporatingoptical camera 110 (such as a smartphone) may request that the userundergo baseline testing, where the optical camera 110 analyzes anddetects superficial arteries in the user's face while the user islooking at optical camera 110. In particular embodiments, one or more ofthe superficial arteries may be selected based on a signal-to-noiseratio (SNR) associated with the BVP for each of the superficialarteries. As an example and not by way of limitation, the SNR associatedwith the BVP may be computed by determining the energy in a dominantelectromagnetic frequency and its second harmonics as compared to therest of the signal measured at the ROI. Once arteries are mapped for agiven user, then after adjusting for face position or lightingartifacts, the arteries with the highest SNRs may be selected.

FIG. 3 illustrates an example calculation of cardiological metrics. Theblood pressure (BP) differs based on the amount of blood which flowsthrough the body at a given instant of time. In particular embodiments,the heart rate or RR-interval may be calculated at an artery locationusing a blood-volume pulse method. Once an artery is located, asdescribed above, the HR may be determined by analyzing a signaltransmission model of blood flow along a length of the selected artery.In particular embodiments, cardiological metrics may be calculated usingthe blood volume and stiffness associated with the artery. The bloodvolume may be determined based on calculating the area under the curve(AUC) of the PPG signal. Camera-based pulse wave transit time (PWTT)measurements may detect changes in blood volume, which may be used instructural analysis of the associated artery. PWTT is measure of theamount of time taken by a pulse wave to travel between two points in acirculatory system. As an example and not by way of limitation, thestiffness associated with an artery may be determined based on the PWTT.In the example of FIG. 3, the PWTT and pulse wave velocity may becalculated based on a time difference from an initial time T₁ to a finaltime T₂ for blood to flow along a length L of an artery, as illustratedby equations (1) and (2) below.

PWTT=T2−T1  (1)

Pulse Wave Velocity=L/(T2−T1)  (2)

FIG. 4 illustrates an example method for calculating cardiologicalmetrics for a user. As described below, cardiological metrics of theuser may be evaluated by correlating the features of brain-wave activityand cardiac activity to particular emotions. The example method 400 maybe deployed on one or more electronic devices, such as a mobile phone,tablet computer, mobile devices, fixed-location computing devices,and/or network or cloud deployed systems, having suitable communication,processing, sensing facilities, or any combination thereof. As anexample and not by way of limitation, a processor or circuitry mayembody the method (e.g., one or more computer processors may becommunicatively coupled to a data storage device that storesinstructions executable by the one or more computer processors toperform operations of example method 400 for cardiological metricevaluation).

The method 400 begins at step 410, where an optical camera associatedwith a computing system captures images of a user in real-time. Inparticular embodiments, the optical camera is attached to or integratedwith the computing system. At step 420, one or more regions of interestcorresponding to one or more superficial arteries of the user areidentified. In particular embodiments, the identification of thesuperficial arteries is based on the SNR of photoplethysmogram (PPG)data obtained from the captured images. At step 430, the computingsystem may measure, based on the PPG data, blood volume pulse (BVP)signals. At step 440, based on the measured BVP signals, the computingsystem can compute one or more cardiological metrics for the user.

Particular embodiments may repeat one or more steps of method 400 ofFIG. 4, where appropriate. Although this disclosure describes andillustrates particular steps of the method of FIG. 4 as occurring in aparticular order, this disclosure contemplates any suitable steps of themethod of FIG. 4 occurring in any suitable order. Moreover, althoughthis disclosure describes and illustrates an example method forcalculating cardiological metrics for a user, including the particularsteps of the method of FIG. 4, this disclosure contemplates any suitablemethod for calculating cardiological metrics for a user including anysuitable steps, which may include all, some, or none of the steps of themethod of FIG. 4, where appropriate. Furthermore, although thisdisclosure describes and illustrates particular components, devices, orsystems carrying out particular steps of the method of FIG. 4, thisdisclosure contemplates any suitable combination of any suitablecomponents, devices, or systems carrying out any suitable steps of themethod of FIG. 4.

In particular embodiments, metrics associated with respiration may bemeasured using remote PPG (rPPG) to perform concurrent analysis ofmultiple arteries located on or in a vicinity of the nose. Theparasympathetic nervous system is activated by exhalation duringrespiration. By measuring the movement of blood through the nasalarteries, a number of respiration metrics including inhalation andexhalation time may be determined by measuring blood volume change aswell as the periodic motion of the blood flowing through the respectiveartery. The respiratory rate of the user may be computed using acomposite BVP signal. In particular embodiments, the composite BVPsignal is a signal based on concurrent measurements of BVP change acrossmultiple superficial nasal arteries. The mechanics of respiration mayinduce periodic head motions, that in turn modulate the signal (e.g.,through reflection) from the nasal arteries. In particular embodiments,the effect of the periodic head motions on the resultant composite BVPsignal may be compensated using a measurement of BVP change across anon-nasal artery. As an example and not by way of limitation, thecomposite BVP signal may be refined using a forehead artery. Thearteries of the forehead are free from emotion expressions that maymodulate light reflection from the face. Although this disclosureillustrates and describes measuring particular cardiological andrespiration metrics using particular types of sensors, this disclosurecontemplates the use any suitable types of sensors for measuringcardiological and respiratory metrics.

As described above, one or more cardiological measurements may becalculated based on the physiologically plausible or analyzablesub-segments of the PPG waveform. In particular embodiments,physiologically plausible or analyzable sub-segments may be defined assub-segments that satisfy a threshold SNR. Features (e.g., peaks, zerocrossings, or troughs) of the PPG waveform may be used to find one ormore analyzable segments for extracting or calculating cardiological andrespiratory metrics. In particular embodiments, analyzable segments maybe obtained by analyzing the signal-to-noise ratio (SNR) of the PPGsignal with respect to a baseline signal and then computing a noisequotient of the test signal with respect to the dominant signal and itsharmonics, depending on whether the underlying processes being captureduses a reflected chrominance signal or a pulmonary blood volume (PBV)signal. In other embodiments, the signal components may be obtained froma theoretical model or from a database and expected frequencycomponents. As described below, a PPG signal profile may be computed toguide pathological cases. As an example and not by way of limitation, inpeople with arrhythmias, the baseline signal, even if completely noisefree, may falsely appear to be a highly corrupted signal.

The PPG waveform extracted from images from the camera is a time seriesthat may be segmented into analyzable sub-segments for finding the HR.In particular embodiments, one or more pre-defined, physiological signalcharacteristics may be used to identify the analyzable sub-segments ofthe PPG waveform. In particular embodiments, the BVP and pulse timingmay be compared with the expected behavior of a respiratory profile.Valid segments should correspond to characteristics expected out of ameasured BVP, such as for example consistency between timing, amplitude,morphology, or respiratory sinus arrhythmia (RSA). Valid cardiac pulsesmay be extracted through analysis of respiratory variations of PPGwaveforms and to identify the PPG waveforms with inconsistent orimplausible RR interval (where “RR” refers to the time betweenheartbeats) or amplitude. Physiologically plausible pulses may conformwith characteristics that are expected from BVP measurements. As anexample and not by way of limitation, during exhalation the BVPprogressively increases while the RR intervals also progressivelyincreases, and the converse is the case during inhalation. Theco-occurrence or high covariance between these signals may be anindication of the physiological plausibility of the given waveform.

As another example, analyzable sub-segments of the PPG waveform maycorrespond to a RR-interval that is consistent with the dominantfrequency in the fast Fourier transform (FFT) of the signal. Inparticular embodiments, the overall time series may be refined usingcontextual analysis of frequency transform over short time seriessegments to remove pulses that lead to a HR calculation that isphysiologically implausible. In addition, pulse origins of theanalyzable time segments must not happen at physiologically implausiblepoints.

Additionally, the profile of the BVP should be consistent with apre-defined blood pulse, where the rising part (the systolic part) isless than the diastolic part (the decaying part of the pulse). Foranalyzable segments, the 90% width of the blood pulse should beapproximately equal to the RR-interval for cases where there is nopulsatile activity between two signals. As described above, pre-definedsignal characteristic is used to identify analyzable sub-segments of thePPG signal that are consistent with a pre-determined respiratory profilethat is physiologically plausible. As an example and not by way oflimitation, the PPG waveform may be analyzed to determine whether themeasured pulse volume measurement progressively increases while Rwave-to-R wave (RR) intervals progressively increase during exhalation.Conversely, analyzable sub-segments should have a pulse volumemeasurement that progressively decreases while RR-intervalsprogressively decrease during inhalation. In addition, analyzablesub-segments of the PPG signal may be confirmed by determining whether acorresponding systolic part of the signal is less than a correspondingdiastolic part of the same signal.

As described above, the PPG waveform extracted from data captured by acamera includes a red, green, and blue channel component. Visible lightat different wavelengths, corresponding to different colors of thevisible spectrum (e.g., red, green, or yellow), penetrates skin tovarying depths depending on its wavelength. In particular embodiments,the PPG waveform may be analyzed to determine whether the signalsassociated with any of the red, green, or blue channels arephysiologically implausible based on pre-determined criteria. As anexample and not by way of limitation, one such criterion is based on theDC power of the signal reflected from the face of the user. The power ofthe reflected signal is inversely proportional to the amount of lightabsorbed by skin. Human skin may be thought of having two layers, theepidermis and dermis. The epidermis contains melanin, which has apreferential absorption for lower wavelength (blue) light to protecthumans from ultraviolet rays. For this reason, a significant portion ofthe incident blue light does not make it through the epidermis and intothe dermis. Therefore the blue component of visible light has the lowestpenetration depth, but most of the light is not reflected, but insteadis absorbed by the epidermis. Additionally, if blue light penetrates tothe dermis, this blue component interacts with hemoglobin (and othercutaneous pigments) and goes through an additional absorption furtherreducing the amount of reflection. Therefore, the blue component ofvisible light has the lowest reflectance, such that the signal from theblue channel has the lowest total power of the overall signal.

The red component is the only component reflected at all layers of theskin leading to the red component having the highest penetration depthand total power. In particular embodiments, a PPG segment may bediscarded as unreliable or implausible if the total power of the bluecomponent is higher than either the red or green component total power.The DC blue reflectance should be less than the DC reflectance greencomponent, which is less than DC reflectance of the red component. Thus,in analyzable segments of the PPG waveform, the total power of the redcomponents should be greater than the total power of the greencomponents, which should be greater than the total power of the bluecomponents. In particular embodiments, a PPG segment may be discarded asunreliable if it does not conform to this pattern. It should be notedthat these criteria are strictly valid under the assumption that theincident light is “white” (approximately equal amounts of each colorcomponent (e.g., red, green, and blue) is incident to the skin surfaceof the user). For example, if a narrow blue-light source was incident tothe surface of the human skin, then the criterion above may not bevalid. In particular embodiments, this assumption of a “white” lightsource may be tested using the optical camera while performing baselinemeasurements.

Another example criterion is based on the AC component of the signal.The AC component of the signal results from the reflectance variation asblood flows (e.g., from the heart beat of the user) through the ROI. Inparticular embodiments, the captured signal from the camera may bevalidated based on the pre-defined signal characteristics, as describedabove. Blood is the main chromophore in the face and causes strongerattenuation then the melanin, and leads to greater absorption of thegreen component of than the red component. The effect leads to thepulsatile (i.e., AC) power of the green channel to be the largest of the3 components. As an example and not by way of limitation, a segment maybe discarded as unreliable if the pulsatile power of the green componentis lower than the pulsatile power of the red or blue channel components.Variations in both the AC and DC components of the captured signal mayarise from changes in environmental illumination. These variations fromenvironmental illumination may be compensated (e.g., throughmeasurements of a forehead artery, as described above) or thecorresponding segments may be flagged as invalid.

Another example criteria to validate segments of the PPG waveform isbased on the covariance of the signal. In particular embodiments, thecaptured signal from the camera may be validated based on whether thered, green, and blue channel components are co-variant for a particularsignal, such that pulses in three channel components are time aligned.As described above, each color channel is mainly modulated by the flowand light absorption from blood. Segments of the signal where thecomponents from the three channels do not have high covariance may berejected and not used in calculating the relevant cardiological orrespiration metrics.

In particular embodiments, segments of the PPG waveform may be rejectedon the discrete Fourier transform (DFT) power of the segments. As anexample and not by way of limitation, the DFT peak should be consistentwith RR or RR-deltas. In particular embodiments, the SNR of the timesegments may be considered when comparing the RR values to the DFT peak.As an example and not by way of limitation, particular embodiments checkfor a suitably high SNR between the narrowband frequency component andthe noise floor. If SNR is low, the current window (i.e., PPG samplebeing analyzed) is discarded. Segments may be discarded or correctedbased on observing the values within a time window. In particularembodiments, time segments may be rejected based on a threshold formaximum acceptable median of windowed covariance between normalizedmean-centered RGB channels.

As described above, the analyzable segments of the PPG waveform may beused to calculate an average HR or HR range of the user. In a firststep, analyzable segments may be identified using co-variance analysis.In particular embodiments, this step may occur during an initializationperiod. The second step involves subsequent tracking of arteries ofinterest using images collected from the camera, as described above. Ina third step, high valued impulses (e.g., with sharpedges/discontinuities) may be removed or ignored in the HR calculation.In a fourth step, signal transformation may be performed using pixelquotient in log space or blood volume pulse. In a fifth step, signalfeatures (e.g., peak, trough, zero-crossings, signal profile, SNR, etc.)may be used to calculate the RR. In a sixth step, physiologicallyplausible signal or analyzable segments may be identified based on inthe signal R, G, B channels, the total signal power and the totalpulsatile power obtained from the images captured by the camera, asdescribed above. Physiologically plausible or analyzable segments may beidentified using respiration, blood volume (signal amplitude), andtiming variation analysis, as described above. In a next step,implausible pulses may be removed using frequency analysis. Noisysegments in a time series may be identified by comparing the SNR of timeseries with the baseline signal profile. In particular embodiments, RRis determined based on time-domain analysis using proximal pulses, whichis then used to calculate an average HR or HR range. In particularembodiments, the above analysis may be performed using independentcomponent analysis (ICA) of the time series, where the time series isincremented in periodic increments of a pre-determined number of frames,until the mutual information (MI) of the independent components is closeto 0. The HR may be calculated by calculating the FFT of the largestindependent component.

FIG. 5 illustrates an example method for performing heartratemeasurements for a user. The method 500 begins at step 510, where anoptical camera associated with a computing system captures images of auser in real-time. In particular embodiments, the optical camera isattached to or integrated with the computing system. At step 520, thecomputing system may determine a time-series signal for the user basedon the plurality of images. In particular embodiments, the signalincludes one or more segments that are physiologically plausible and oneor more segments that are physiologically implausible. At step 530, thecomputing system may identify one or more of the physiologicallyplausible sub-segments based on one or more pre-defined signalcharacteristics. In particular embodiments, the identification mayinclude calculating the SNR of the time-series and comparing the SNR toa threshold SNR. Sub-segments of the signal with a calculated SNR thatis higher than the threshold SNR may be identified. At step 540, thecomputing system may calculate one or more heartrate measurements basedon the physiologically plausible sub-segments.

Particular embodiments may repeat one or more steps of method 500 ofFIG. 5, where appropriate. Although this disclosure describes andillustrates particular steps of the method of FIG. 5 as occurring in aparticular order, this disclosure contemplates any suitable steps of themethod of FIG. 5 occurring in any suitable order. Moreover, althoughthis disclosure describes and illustrates an example method forperforming heartrate measurements for a user, including the particularsteps of the method of FIG. 5, this disclosure contemplates any suitablemethod for performing heartrate measurements for a user including anysuitable steps, which may include all, some, or none of the steps of themethod of FIG. 5, where appropriate. Furthermore, although thisdisclosure describes and illustrates particular components, devices, orsystems carrying out particular steps of the method of FIG. 5, thisdisclosure contemplates any suitable combination of any suitablecomponents, devices, or systems carrying out any suitable steps of themethod of FIG. 5.

The cardiovascular system is regulated by the ANS, that includes thesympathetic nervous system (SNS) and parasympathetic nervous systems(PSNS). Sympathovagal balance (SVB) is defined as the equilibrium pointbetween the SNS and the PSNS. The rhythm of the heart is controlled bythe sinoatrial (SA) node, which is modulated by both the sympathetic andparasympathetic branches of the autonomic nervous system. The heartreceives its neural input through parasympathetic and sympatheticganglia and lateral grey column of the spinal column. Sympatheticactivity tends to increase heart rate, while parasympathetic activitytends to decrease heart rate. In particular embodiments, SVB-basedevaluation of emotions may determine a homeostatic equilibrium where thesympathetic outflow and PSNS (e.g., vagal nerve) outflow are in stablebalance. Accordingly, SVB may be determined by analyzing relativedominance and balance between the SNS and PSNS (e.g., betweenparasympathetic outflow and sympathetic outflow). This approach islinear in nature, which makes it very stable to noise (e.g., which tendsto be non-linear in nature).

FIG. 6A illustrates an example of baseline data histogram. Asillustrated in the example of FIG. 6A, the x-axis of baseline datahistogram 600 corresponds to heart rate fluctuations (e.g., RR-deltas orRR interval differences between successive heart beats), and a y-axis ofbaseline data histogram 600 corresponds to a frequency or number oftimes particular heart rate fluctuations occur for a set of baselinedata. A portion of a set of measurement data that is on one side of thebaseline SVB is used to determine a number of ANS activity markerswithin a first portion of cardiac activity data. A HRV characteristicthat is a ratio involving the number of ANS activity markers within thefirst portion and the number of ANS activity markers within a secondportion of the cardiac activity data is calculated. The SVB is definedby the point of 50:50 distribution of baseline data histogram 600 suchthat 50% of the data points of the set of baseline data are to the leftof the SVB and 50% of the data points of the set of baseline data are tothe right of the SVB. HRV data is used to determine changes in the SVB.HRV describes the variations between consecutive heartbeats. Usingbaseline data histogram 600, the SVB corresponds to a HRV zonerepresentative of a “normal” (e.g., non-stressed) state. The analysis ofSVB described treats the SNS and PSNS with equal weightage (e.g.,because both may be useful in analyzing the ANS arousal) and uses theRR-deltas.

In particular embodiments, to determine SVB, stress or arousal isdefined as increased dominance of the SNS in the RR-delta histogramH_(t) of test data t, and the data points on the histogram correspond tothe difference between a length of adjacent RR intervals in a set oftest data. In particular embodiments, a ratio of a number of eventsrelatively mapped to SNS with a number of events relatively mapped toPSNS is computed. Measurements of HRV may be based on time-domainanalysis or frequency domain analysis, which both have numerousshortcomings. As an example, frequency domain analysis methods such asthose based on FFT, are not very suitable for implementation on mobileplatforms since they are extremely sensitive to the noise artifacts, andrequire rather long measurement time periods. Time domain methods, suchas root mean square of successive heartbeat interval differences(RMSSD), standard deviation of NN (beat-to-beat) intervals (SDNN), andthe proportion of the number of pairs of successive NNs that differentby more than 50 milliseconds divided by the total number of NNs (pNN50),are frequently used to analyze the instantaneous heart-rate signal.Given a baseline SVB of the user, and an RR-delta histogram H_(t), theSVB may be computed as a median of all RR-deltas, a mean value of allRR-deltas, 50% of RMSSD, computed over H_(t), 50% of SDNN, computed overH_(t), other relevant values, or any combination thereof.

FIG. 6B illustrates an example of a test-data histogram 610 in which theSVB is shifted to the left, and FIG. 6C illustrates an example of atest-data histogram 620 in which the SVB is shifted to the right. Asillustrated in the example of FIG. 6B, similar to FIG. 6A, an x-axis oftest data histogram 610 corresponds to heart rate fluctuations (e.g.,RR-deltas), and a y-axis of test data histogram 610 corresponds to anumber of times of the heart rate fluctuations for a set of test data.In particular embodiments, the first and second portions of the set ofmeasurement data are selected based on a baseline SVB value (e.g., auser-specific baseline SVB value) that divides a histogramrepresentation of the set of measurement data into the first and secondportions. When the heart rate fluctuations for the set of test data onaverage results in the shifting of balance from histogram A1 (e.g.,corresponding to the baseline data) toward a left direction to histogramA2, or in other words, the number of measurements of heart ratefluctuations of histogram A2 that are to the left of the SVB (e.g., asdetermined based on the baseline data) are greater than the number ofmeasurements of heart rate fluctuations of histogram A2 that are to theright of the SVB, then it is determined that the user's statecorresponds to a HRV zone representative of a “high arousal” state.Explained another way, as shown in FIG. 6B, given a total area A, thearea of A2 over A (e.g., the area under the curve of A2) is determinedto be less than the area of A1 over A (e.g., the area under the curve ofA1) when the histogram shifts to the left and it is determined that theuser's state corresponds to a HRV zone representative of a “higharousal” state or increased SVB.

On the other hand, as shown in FIG. 6C, similar to FIG. 6A, an x-axis oftest-data histogram 620 corresponds to heart rate fluctuations (e.g.,RR-deltas), and a y-axis of test-data histogram 620 corresponds to anumber of times of the heart rate fluctuations for a set of test data.When the heart rate fluctuations for the set of test data on averageresults in the shifting of balance from histogram A1 (e.g.,corresponding to the baseline data) toward a right direction tohistogram A2, or in other words, the number of measurements of heartrate fluctuations of histogram A2 that are to the right of the SVB(e.g., as determined based on the baseline data) is greater than thenumber of measurements of heart rate fluctuations of histogram A2 thatare to the left of the SVB, then it is determined that the user's statecorresponds to a HRV zone representative of a “low arousal” state.Explained another way, as illustrated in the example of FIG. 6C, given atotal area A, the area of A2 over A (e.g., the area under the curve ofA2) is determined to be greater than the area of A1 over A (e.g., thearea under the curve of A1) when the histogram shifts to the right andit is determined that the user's state corresponds to a HRV zonerepresentative of a “low arousal” state.

Particular embodiments depend on dividing the RR-delta histogram, H_(t),based on the notion of contextually appropriate SVB. As an example andnot by way of limitation, if the context is all the cyclic componentsresponsible for variability in the period of recording, then the ratiofor SVB may be evaluated via the SDNN model.

FIG. 7 illustrates an example diagram of the circumplex model ofemotion. The circumplex model of emotions 700 maps human emotions into atwo-dimensional circular space 702 containing arousal and valencedimensions. Levels of arousal are mapped to the vertical axis and thevalence is mapped to the horizontal axis of circular space 702. In thecircumplex model 700, the valance corresponds to a type of emotion(e.g., positive or negative) while the arousal corresponds to anintensity of the emotion. The center of circular space 702 representsneutral valence and arousal levels. In this model, emotional states canbe represented at any level of valence and arousal, or at a neutrallevel of one or both of these factors. In addition, circular space 702may be partitioned in four quadrants I-IV that each correspond to agrouping of emotional states with similar levels of arousal and valance.As an example and not by way of limitation, quadrant I emotionscorrespond to positive valance and high-arousal emotional states. In theillustrated embodiment, emotional states of quadrant I comprise alert,excited, elated, and happy. In order of increasing valence, theseemotional states of quadrant I are alert, excited, elated, and happy. Inorder of increasing arousal, these emotional states of quadrant I arehappy, elated, excited, and alert.

Quadrant II emotions correspond to negative valence and high-arousalemotional states. In the illustrated embodiment, emotional states ofquadrant II include tense, nervous, stressed, and upset. In order ofincreasing valence, these emotional states of quadrant II are upset,stressed, nervous, and tense. In order of increasing arousal, theseemotional states of quadrant II are upset, stressed, nervous, and tense.

Quadrant III emotions correspond to a negative valance and low-arousalemotional states. In the illustrated embodiment, emotional states ofquadrant III include sadness, depression, lethargy, and fatigue. Inorder of increasing valence, these emotional states of quadrant III canbe listed as sadness, depression, lethargy, and fatigue. In order ofincreasing arousal, these emotional states of quadrant III are fatigue,lethargy, depression, and sadness.

Quadrant IV emotions correspond to positive and low-arousal emotionalstates. In the illustrated embodiment, emotional states of quadrant IVcomprise calm, relaxed, serene, and content. In order of increasingvalence, these emotional states of quadrant IV are calm, relaxed,serene, and content. In order of increasing arousal, these emotionalstates of quadrant IV are calm, relaxed, serene, and content. It will beappreciated that additional emotions can be included, or one or moreemotions of FIG. 7 can be omitted or renamed, in alternative exampleembodiments.

Positive emotions may last for just 10-15 seconds, and so the detectionof positive emotions must be performed in a relatively short period oftime. In particular embodiments, cardiology-centric analysis todetermine emotions of a user may be based on a predictive model of shortterm emotions by combining linear and non-linear models. As describedbelow, an estimate of the emotions of a user may computed or determinedbased at least in part on cardiological-based data (e.g., heart rate orSVB) and in conjunction with neuroanatomical or vestibular-somaticreflex data (e.g., posture, gesture, or laterality) measured via anoptical camera, such as optical camera 110. In particular embodiments,vestibular-somatic reflex data and simultaneous concordance ofcardiological signals may be used to detect a user's emotion, where thecardiological measurements may be determined using a composite model oflinear and nonlinear approaches, as described above. As an example andnot by way of limitation, the cardiological data may be measured using alinear model for detecting emotion based on the user's sympathovagalbalance (SVB). As another example, the cardiological data may bemeasured using a non-linear model of HR dynamics that is based onheartrate dynamics that converge in approximately 12 seconds. Inparticular embodiments, the non-linear model is based on aprobabilistic-point process non-linear model that may be personalizedfor the user. This approach models a probability function of the nextheartbeat given past events. In particular embodiments, the probabilityfunction may be fully parametrized and consider up to cubicauto-regressive Wiener-Volterra relationship to model its first ordermoment. In other words, these representations may be thought of asconvolution of the input signal with a system's impulse response, alongwith a series of nonlinear terms that contain products of increasingorder of the input signal with itself.

In particular embodiments, neurology-centric data may be measured byanalyzing the neurological signal that is measured using a facial actioncoding system (FACS) approach. The FACS approach works by interpretingmuscular activities behind the facial expression which is achieved bycoding different facial movements into Action Units (AU). These actionunits are based on the underlying muscular activity that produces thegiven, and typically, momentary changes in the facial expression. Anexpression is then recognized by identifying the action unit or thecombination of action units which are behind a particular expression.Some of the key parameters used by FACS include eye brows, lip width,mouth, teeth visibility, chin lines, nasolabial lines, nose lines,and/or cheek-based impression of smile lines extracted from the imagescaptured using the camera. FACS is based on more than 30 individualaction units that are behind a given facial expression of the user. Inparticular embodiments, a machine-learning (ML) model, such as forexample neural networks, hidden Markov models (HMM), or SVM (supportvector machines) may be employed to train the interpretation of theaction units extracted from the images of the user captured by thecamera. Since genders show emotion differently, the adjusted ML modelmay be adjusted for gender.

Neuroanatomical or vestibular-somatic reflex (e.g., posture, gesture, orlaterality) data may be captured using the optical camera. In particularembodiments, the images captured by the optical camera may be analyzedto determine changes in laterality or posture of the user. As an exampleand not by way of limitation, the computing system may determine whethera user is experiencing a positive or negative emotion based onlaterality since negative emotions may be shown more prominently on leftside of the face, while positive emotions may be more prominently shownon the right side of the face. As another example, the computing systemmay determine whether a user is experiencing a positive or negativeemotion based on body posture since bad body posture may be associatedwith negative emotions, while good body posture may be associated withpositive emotions.

Cardiological signals may be determinative of short-term positiveemotions if results of both the SVB linear model and heartrate dynamicsnon-linear model are concurrently sensed and are consistent with aparticular emotion. Affective elicitations longer than 6 seconds allowthe prefrontal cortex to encode the stimulus information and transmit itto other areas of the central autonomic network (CAN) from thebrainstem. Therefore, positive emotions should have a duration of atleast 6 seconds plus at least an additional respiratory cycle or more tobe detected. The embodiments herein enable detection of an emotionalprocess that lasts somewhere in the vicinity of 10-15 seconds. Inparticular embodiments, a real-time determination (e.g., withinapproximately 15 seconds) that the user is experiencing a short-termpositive emotion may be further verified using current contextualinformation associated with the user, including an environment of theuser, an activity the user is engaged in, and/or detected physicalattributes of the user (e.g., user posture) based at least in part onneurology-centric data, as described above.

In particular embodiments, a computing system may include anemotion-evaluation module. Herein, reference to a module may refer toself-contained computer code or executable that is configured to performa discrete function. Furthermore, a module may be a dedicated electroniccircuit configured to perform the discrete function. The covariance ofthe RBG channels may be analyzed. In particular embodiments, thecomputing system computes the HRV-based SVB by analyzing a histogram ofHRV measured at pre-determined time intervals (e.g., every 30 seconds),as illustrated in the example of FIGS. 6A-C. As described above, SVB isdefined as the equilibrium point between the SNS and the PSNS, thereforethe time requirement for accurate SVB calculations is dependent on thestabilization times for the SNS and PSNS measurement. As an example andnot by way of limitation, PSNS measurements may take approximately 2 to3 seconds to stabilize whereas SNS measurements may take approximately25 seconds. Histograms with larger bins (or longer measurement times)compensate for latency of the system, while smaller bins (or smallermeasurement times) may lead to inaccurate SVB calculations. In anexample embodiment, SVB and HR are not binary but rather quantify theintensity of SVB and HR activation. Accordingly, the computedintensities can be used to increase the resolution and number ofemotional states that can be estimated.

It can take a certain length of time before the brain can process anemotion by higher-order centers of the brain. For example, this timeperiod can be 6 seconds or more. Given the fact that a typical breathingcycle of users is around 4 second per breath which leads to RSA-basedvariations in HR and HRV and SVB, example embodiments may use apre-determined time window (e.g., 10 seconds) to affirm the covarianttrend from the signals from cardiological data (e.g., from PPG waveform)and neurology-centric data captured from the camera.

Classification is the correlation of an output to a given input (e.g.,emotion to neurology-centric and cardiological data). For example, aparticular feature vector (e.g., an ordered list of features) may becorrelated to a particular emotion of the user. Classification may beperformed using a predictor function that is constructed using a set of“training” data that includes an input vector and an answer vector oflabeled data. A machine-learning classifier algorithm may combine (e.g.,through a dot product) the input vector with one or more weights toconstruct a predictor function to best fit the input vector to theanswer vector. As an example and not by way of limitation,classification algorithms may include support vector machine (SVM),Naive Bayes, Adaptive Boosting (AdaBoost), Random Forest, GradientBoosting, K-means clustering, Density-based Spatial Clustering ofApplications with Noise (DBSCAN), or Neural Network algorithms.

In particular embodiments, during offline training, the training datamay be obtained from the neurological (e.g., FACS-based) andcardiological (e.g., HR and SVB) data from a number of users that iscaptured using the camera and the corresponding emotion of the user atthe time the data is being measured. As an example and not by way oflimitation, the input vector may be a vector of the extractedneurocardiological features and the answer vector may be thecorresponding emotion (e.g., happiness) as reported by the user. As anexample and not by way of limitation, training data for emotionevaluation may be images of the users captured while watching a content(e.g., a sports game, TV show, or movie). In particular embodiments, theoutput vector of the machine-learning classifier may be one or morequadrants or emotions of circumplex model 700 and the output vector maybe compared to the answer vector of labeled data (e.g., theuser-provided emotion) to “train” the predictor function of themachine-learning classifier.

Facial expressions reflect the effect of brain on the PNS (peripheralnervous system) and the HR/HRV reflect the state of the heart. Inparticular embodiments, the emotion-evaluation module may receivecardiological and neurological data from images of the user captured bythe camera. In particular embodiments, the emotion-evaluation module maybe configured to determine emotions of the user based on the covarianceof the cardiological and neurological data over a significant period oftime (e.g., ˜5 or more seconds). In particular embodiments, theemotion-evaluation module may perform an estimation of emotional stateof the user during a period of time based on the cardiological data.Moments in time corresponding to pre-determined thresholds ofcardiological and neurological data may be used initiate the evaluationof emotions of the user. As an example and not by way of limitation,moments in time corresponding to thresholds (e.g., HRV rising above 60and a particular facial expression) may be captured and associated witha particular emotion (e.g., elation or excitement). Theemotion-evaluation module may include an emotional-state comparisonmodule. In particular embodiments, the emotion-evaluation module mayidentify emotions based on determining a particular quadrant of thecircumplex model of emotion 700 that corresponds to the cardiologicaland neurological data. As an example and not by way of limitation, theemotion-evaluation module may compute that SVB and HR both increasing atsubstantially the same time corresponds to quadrant I (upper-rightquadrant) of the circumplex model 700, SVB decreasing and HR increasingcorresponds to quadrant II (upper left quadrant), SVB and HR bothdecreasing corresponds to quadrant III (lower-left quadrant), and SVBincreasing and HR decreasing corresponds to quadrant IV (lower-rightquadrant).

In particular embodiments, the determined emotions of the user may berefined using the neurological data (e.g., posture & activitycontext-based refinement). As an example and not by way of limitation,decreased SVB and increased HR with a concomitant facial expression maycorrespond to anger. As another example, decreased SVB and increased HRwith a concomitant facial expression may correspond to nervousness. Inparticular embodiments, the emotion-evaluation module refines theemotion of the user within the particular quadrant of the circumplexmodel 700 based on the emotion of the user determined by neuroanatomicaldata (e.g., posture or gestures captured by the camera).

As described above, SVB decreasing and HR increasing is indicative ofemotions in quadrant II (e.g., upset, stress, nervous, or tension) ofcircumplex model 700. In particular embodiments, the analysis of angeris refined through detection of a concomitant detection of bad postureor emotion displayed on the left side of the face. As an example and notby way of limitation, the emotional-state comparison module may estimatethe user is feeling elated based determining both SVB and HR increasing,a corresponding facial expression, and at the same time detecting thatthe user has good posture or the displaying emotion on the right side ofthe face. As described above, all of the signals necessary to make thesedeterminations may be captured by a signal optical sensor, such asoptical camera 110. As used herein, increases and decreases in SVB, andHR levels are relative to baseline values. In particular embodiments, ifthe trend in the coherence of the neurological and cardiological (interms of SVB or HR or both) data holds for more than a predeterminedamount of time (e.g., 10 seconds), then the computing system may outputthe estimated emotional state.

FIG. 8 illustrates an example method for emotion evaluation. Asdescribed below, the emotions of the user may be evaluated bycorrelating the features of brain-wave activity and cardiac activity toparticular emotions. In particular embodiments, the method 800 may beperformed by an emotion-evaluation system. The method 800 begins at step810, where a single sensor captures images of a user in real time. Atstep 820, a computing system determines in real time, based on thecaptured images, one or more short-term cardiological signals of theuser during a period of time. In particular embodiments, at step 830,the computing system may estimate in real-time, based on thecardiological signals, a first short-term emotional state of the user.In particular embodiments, the estimate of the first short-termemotional state is based on the calculated SVB and HR from a PPGwaveform extracted from the captured images. In particular embodiments,reference to short-time corresponds to a time period of approximately 15seconds. At step 840, the computing system may determine in real-time,based on the captured images, one or more short-term neurologicalsignals of the user during the period of time. At step 850, thecomputing system estimates in real-time, based on the neurologicalsignals, a second short-term emotional state of the user. In particularembodiments, the second short-term emotional state is based onclassification of action units, described above, using FACS. At step860, the computing system may compare the first estimated emotionalstate to the second estimated emotional state. At step 870, in responseto a determination that the first estimated emotional state correspondsto the second estimated emotional state, determine, based on the firstestimated emotional state and second estimated emotional state, ashort-term emotion of the user during the period of time.

Particular embodiments may repeat one or more steps of method 800 ofFIG. 8, where appropriate. Although this disclosure describes andillustrates particular steps of the method of FIG. 8 as occurring in aparticular order, this disclosure contemplates any suitable steps of themethod of FIG. 8 occurring in any suitable order. Moreover, althoughthis disclosure describes and illustrates an example method forevaluating emotions of a user, including the particular steps of themethod of FIG. 8, this disclosure contemplates any suitable method forevaluating emotions of a user including any suitable steps, which mayinclude all, some, or none of the steps of the method of FIG. 8, whereappropriate. Furthermore, although this disclosure describes andillustrates particular components, devices, or systems carrying outparticular steps of the method of FIG. 8, this disclosure contemplatesany suitable combination of any suitable components, devices, or systemscarrying out any suitable steps of the method of FIG. 8.

In particular embodiments, the embodiments described herein may be usedto evaluate the cardiac risk of a user. For example, the determinationof atherosclerosis in carotid artery or radial artery may be extremelyimportant in determining the risk of stroke. In particular embodiments,a surrogate marker of blood pressure may be computed using blood volumepulse (BVP) and/or PWTT. As an example and not by way of limitation,determining PWTT across the carotid artery and blood-volume change mayenable measurement of an extent of atherosclerosis of a user sincearterial stiffness is mediated by the ANS via PWTT. In addition, astroke risk may be computed based on a measured PWTT across and BVPchange in the carotid or radial artery compared with the user'sbaseline. Similarly, a cardiovascular (CV) hazard may be identified bycontinuously measuring PWTT across and/or BVP change in one or more ofthe superficial arteries during a time period during which the userawakens at the time of a morning period.

In particular embodiments, artery mapping may be used to authenticate auser. As described above, facial arteries may be mapped based on the SNRalong ROIs of the face of the user using images of the face that arecaptured by the optical camera. In particular, the angular andsupratrochlear arteries are relatively easy to map. This facial-arterialmapping is unique to each user and may serve as a facial signature thatmay be used as a basis to distinguish between different users.Authentication may be performed using layout of key arteries withrespect to facial landmarks (e.g., eyes, nose, ears, or arteries). As anexample and not by way of limitation, this facial signature may becaptured using the optical camera and stored in a computing system. Auser may be authenticated by mapping the key arteries of the user at thetime of authentication and compared to the stored facial signature. Forexample, a user may be authenticated to a device, such as a smartphoneor an ATM containing the optical camera.

Accurate detection of emotions may be highly applicable to many fields,such as for example virtual reality (VR), gaming, entertainment,targeted advertisement, smart home, health, or education. Client system120 may be a computer or smart TV with one or more optical camerasconfigured to measure neurocardiological and neuroanatomical data of auser. In particular embodiments, the content or a setting associatedwith content displayed for the user may be adjusted based on theevaluated emotion of the user. As an example and not by way oflimitation, a server providing content to the user through a clientsystem may provide a video stream in an interactive way based on theemotion of the user from data captured by the optical camera. Forexample, a server may provide a video stream while the optical cameramay evaluate the emotion of the user. The server may dynamically servecontent to evoke a particular emotion. As an example and not by way oflimitation, the server may select one or more frames of a video streamthat evoke a feeling of relaxation (e.g., cats playing with yarn) forthe user and transmit the selected frames for display on a clientdevice. As another example, the client system may pause content if theuser is determined to be stressed, angry, or afraid. While thisdisclosure describes the server or client system as performing a certainaction(s), this disclosure contemplates that either client or serverdevice may determine an appropriate action to take based on anevaluation of a user's emotion.

As another example, the server may transmit a video stream correspondingto scenes of a video game for display on a client system (e.g., acomputer or smart TV). The server may evaluate the emotion of the userduring game play by capturing images of the face of the user and adjustthe difficulty level of the game based on the emotion of the user. Forexample, if the user is expressing frustration during game play, theserver may transmit game content having a lower difficulty level.Conversely, the server may transmit game content corresponding to ahigher difficulty level based on detecting boredom. As another example,an educational system may determine which content is most engaging to auser and may tailor additional educational content based on what haspreviously made the user experience positive emotions while learning.

In particular embodiments, the server may send the user one or morerecommendations for additional content based on the evaluated emotionsof the user while consuming particular content. As an example and not byway of limitation, the server may send a recommendation for similargenres of games (e.g., first-person shooter or social game) based ondetermining that the user was experiencing happiness or excitement basedon facial images captured by the optical camera while playing aparticular game. As another example, the server may send arecommendation for a different genre of movies or television shows(e.g., action or sports) based on determining that the user wasexperiencing depression or fatigue while watching a particular movie ortelevision show.

FIG. 9 illustrates an example computer system 900 according to someembodiments of the invention. In particular embodiments, one or morecomputer systems 900 perform one or more steps of one or more methodsdescribed or illustrated herein. In particular embodiments, one or morecomputer systems 900 provide functionality described or illustratedherein. In particular embodiments, software running on one or morecomputer systems 900 performs one or more steps of one or more methodsdescribed or illustrated herein or provides functionality described orillustrated herein. Particular embodiments include one or more portionsof one or more computer systems 900. Herein, reference to a computersystem may encompass a computing device, and vice versa, whereappropriate. Moreover, reference to a computer system may encompass oneor more computer systems, where appropriate.

This disclosure contemplates any suitable number of computer systems900. This disclosure contemplates computer system 900 taking anysuitable physical form. As example and not by way of limitation,computer system 900 may be an embedded computer system, a system-on-chip(SOC), a single-board computer system (SBC) (such as, for example, acomputer-on-module (COM) or system-on-module (SOM)), a desktop computersystem, a laptop or notebook computer system, an interactive kiosk, amainframe, a mesh of computer systems, a mobile telephone, a personaldigital assistant (PDA), a server, a tablet computer system, or acombination of two or more of these. Where appropriate, computer system900 may include one or more computer systems 900; be unitary ordistributed; span multiple locations; span multiple machines; spanmultiple data centers; or reside in a cloud, which may include one ormore cloud components in one or more networks. Where appropriate, one ormore computer systems 900 may perform without substantial spatial ortemporal limitation one or more steps of one or more methods describedor illustrated herein. As an example and not by way of limitation, oneor more computer systems 900 may perform in real time or in batch modeone or more steps of one or more methods described or illustratedherein. One or more computer systems 900 may perform at different timesor at different locations one or more steps of one or more methodsdescribed or illustrated herein, where appropriate.

In particular embodiments, computer system 900 includes a processor 902,memory 904, storage 906, an input/output (I/O) interface 908, acommunication interface 910, and a bus 912. Although this disclosuredescribes and illustrates a particular computer system having aparticular number of particular components in a particular arrangement,this disclosure contemplates any suitable computer system having anysuitable number of any suitable components in any suitable arrangement.

In particular embodiments, processor 902 includes hardware for executinginstructions, such as those making up a computer program. As an exampleand not by way of limitation, to execute instructions, processor 902 mayretrieve (or fetch) the instructions from an internal register, aninternal cache, memory 904, or storage 906; decode and execute them; andthen write one or more results to an internal register, an internalcache, memory 904, or storage 906. In particular embodiments, processor902 may include one or more internal caches for data, instructions, oraddresses. This disclosure contemplates processor 902 including anysuitable number of any suitable internal caches, where appropriate. Asan example and not by way of limitation, processor 902 may include oneor more instruction caches, one or more data caches, and one or moretranslation lookaside buffers (TLBs). Instructions in the instructioncaches may be copies of instructions in memory 904 or storage 906, andthe instruction caches may speed up retrieval of those instructions byprocessor 902. Data in the data caches may be copies of data in memory904 or storage 906 for instructions executing at processor 902 tooperate on; the results of previous instructions executed at processor902 for access by subsequent instructions executing at processor 902 orfor writing to memory 904 or storage 906; or other suitable data. Thedata caches may speed up read or write operations by processor 902. TheTLBs may speed up virtual-address translation for processor 902. Inparticular embodiments, processor 902 may include one or more internalregisters for data, instructions, or addresses. This disclosurecontemplates processor 902 including any suitable number of any suitableinternal registers, where appropriate. Where appropriate, processor 902may include one or more arithmetic logic units (ALUs); be a multi-coreprocessor; or include one or more processors 902. Although thisdisclosure describes and illustrates a particular processor, thisdisclosure contemplates any suitable processor.

In particular embodiments, memory 904 includes main memory for storinginstructions for processor 902 to execute or data for processor 902 tooperate on. As an example and not by way of limitation, computer system900 may load instructions from storage 906 or another source (such as,for example, another computer system 900) to memory 904. Processor 902may then load the instructions from memory 904 to an internal registeror internal cache. To execute the instructions, processor 902 mayretrieve the instructions from the internal register or internal cacheand decode them. During or after execution of the instructions,processor 902 may write one or more results (which may be intermediateor final results) to the internal register or internal cache. Processor902 may then write one or more of those results to memory 904. Inparticular embodiments, processor 902 executes only instructions in oneor more internal registers or internal caches or in memory 904 (asopposed to storage 906 or elsewhere) and operates only on data in one ormore internal registers or internal caches or in memory 904 (as opposedto storage 906 or elsewhere). One or more memory buses (which may eachinclude an address bus and a data bus) may couple processor 902 tomemory 904. Bus 912 may include one or more memory buses, as describedbelow. In particular embodiments, one or more memory management units(MMUs) reside between processor 902 and memory 904 and facilitateaccesses to memory 904 requested by processor 902. In particularembodiments, memory 904 includes random access memory (RAM). This RAMmay be volatile memory, where appropriate Where appropriate, this RAMmay be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, whereappropriate, this RAM may be single-ported or multi-ported RAM. Thisdisclosure contemplates any suitable RAM. Memory 904 may include one ormore memories 904, where appropriate. Although this disclosure describesand illustrates particular memory, this disclosure contemplates anysuitable memory.

In particular embodiments, storage 906 includes mass storage for data orinstructions. As an example and not by way of limitation, storage 906may include a hard disk drive (HDD), a floppy disk drive, flash memory,an optical disc, a magneto-optical disc, magnetic tape, or a UniversalSerial Bus (USB) drive or a combination of two or more of these. Storage906 may include removable or non-removable (or fixed) media, whereappropriate. Storage 906 may be internal or external to computer system900, where appropriate. In particular embodiments, storage 906 isnon-volatile, solid-state memory. In particular embodiments, storage 906includes read-only memory (ROM). Where appropriate, this ROM may bemask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM),electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM),or flash memory or a combination of two or more of these. Thisdisclosure contemplates mass storage 906 taking any suitable physicalform. Storage 906 may include one or more storage control unitsfacilitating communication between processor 902 and storage 906, whereappropriate. Where appropriate, storage 906 may include one or morestorages 906. Although this disclosure describes and illustratesparticular storage, this disclosure contemplates any suitable storage.

In particular embodiments, I/O interface 908 includes hardware,software, or both, providing one or more interfaces for communicationbetween computer system 900 and one or more I/O devices. Computer system900 may include one or more of these I/O devices, where appropriate. Oneor more of these I/O devices may enable communication between a personand computer system 900. As an example and not by way of limitation, anI/O device may include a keyboard, keypad, microphone, monitor, mouse,printer, scanner, speaker, still camera, stylus, tablet, touch screen,trackball, video camera, another suitable I/O device or a combination oftwo or more of these. An I/O device may include one or more sensors.This disclosure contemplates any suitable I/O devices and any suitableI/O interfaces 908 for them. Where appropriate, I/O interface 908 mayinclude one or more device or software drivers enabling processor 902 todrive one or more of these I/O devices. I/O interface 908 may includeone or more I/O interfaces 908, where appropriate. Although thisdisclosure describes and illustrates a particular I/O interface, thisdisclosure contemplates any suitable I/O interface.

In particular embodiments, communication interface 910 includeshardware, software, or both providing one or more interfaces forcommunication (such as, for example, packet-based communication) betweencomputer system 900 and one or more other computer systems 900 or one ormore networks. As an example and not by way of limitation, communicationinterface 910 may include a network interface controller (NIC) ornetwork adapter for communicating with an Ethernet or other wire-basednetwork or a wireless NIC (WNIC) or wireless adapter for communicatingwith a wireless network, such as a WI-FI network. This disclosurecontemplates any suitable network and any suitable communicationinterface 910 for it. As an example and not by way of limitation,computer system 900 may communicate with an ad hoc network, a personalarea network (PAN), a local area network (LAN), a wide area network(WAN), a metropolitan area network (MAN), or one or more portions of theInternet or a combination of two or more of these. One or more portionsof one or more of these networks may be wired or wireless. As anexample, computer system 900 may communicate with a wireless PAN (WPAN)(such as, for example, a BLUETOOTH WPAN), a WI-FI network, a WI-MAXnetwork, a cellular telephone network (such as, for example, a GlobalSystem for Mobile Communications (GSM) network), or other suitablewireless network or a combination of two or more of these. Computersystem 900 may include any suitable communication interface 910 for anyof these networks, where appropriate. Communication interface 910 mayinclude one or more communication interfaces 910, where appropriate.Although this disclosure describes and illustrates a particularcommunication interface, this disclosure contemplates any suitablecommunication interface.

In particular embodiments, bus 912 includes hardware, software, or bothcoupling components of computer system 900 to each other. As an exampleand not by way of limitation, bus 912 may include an AcceleratedGraphics Port (AGP) or other graphics bus, an Enhanced Industry StandardArchitecture (EISA) bus, a front-side bus (FSB), a HYPERTRANSPORT (HT)interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBANDinterconnect, a low-pin-count (LPC) bus, a memory bus, a Micro ChannelArchitecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, aPCI-Express (PCIe) bus, a serial advanced technology attachment (SATA)bus, a Video Electronics Standards Association local (VLB) bus, oranother suitable bus or a combination of two or more of these. Bus 912may include one or more buses 912, where appropriate. Although thisdisclosure describes and illustrates a particular bus, this disclosurecontemplates any suitable bus or interconnect.

Herein, a computer-readable non-transitory storage medium or media mayinclude one or more semiconductor-based or other integrated circuits(ICs) (such, as for example, field-programmable gate arrays (FPGAs) orapplication-specific ICs (ASICs)), hard disk drives (HDDs), hybrid harddrives (HHDs), optical discs, optical disc drives (ODDs),magneto-optical discs, magneto-optical drives, floppy diskettes, floppydisk drives (FDDs), magnetic tapes, solid-state drives (SSDs),RAM-drives, SECURE DIGITAL cards or drives, any other suitablecomputer-readable non-transitory storage media, or any suitablecombination of two or more of these, where appropriate. Acomputer-readable non-transitory storage medium may be volatile,non-volatile, or a combination of volatile and non-volatile, whereappropriate.

Herein, “or” is inclusive and not exclusive, unless expressly indicatedotherwise or indicated otherwise by context. Therefore, herein, “A or B”means “A, B, or both,” unless expressly indicated otherwise or indicatedotherwise by context. Moreover, “and” is both joint and several, unlessexpressly indicated otherwise or indicated otherwise by context.Therefore, herein, “A and B” means “A and B, jointly or severally,”unless expressly indicated otherwise or indicated otherwise by context.

The scope of this disclosure encompasses all changes, substitutions,variations, alterations, and modifications to the example embodimentsdescribed or illustrated herein that a person having ordinary skill inthe art would comprehend. The scope of this disclosure is not limited tothe example embodiments described or illustrated herein. Moreover,although this disclosure describes and illustrates respectiveembodiments herein as including particular components, elements,feature, functions, operations, or steps, any of these embodiments mayinclude any combination or permutation of any of the components,elements, features, functions, operations, or steps described orillustrated anywhere herein that a person having ordinary skill in theart would comprehend. Furthermore, reference in the appended claims toan apparatus or system or a component of an apparatus or system beingadapted to, arranged to, capable of, configured to, enabled to, operableto, or operative to perform a particular function encompasses thatapparatus, system, component, whether or not it or that particularfunction is activated, turned on, or unlocked, as long as thatapparatus, system, or component is so adapted, arranged, capable,configured, enabled, operable, or operative. Additionally, although thisdisclosure describes or illustrates particular embodiments as providingparticular advantages, particular embodiments may provide none, some, orall of these advantages.

What is claimed is:
 1. One or more computer-readable non-transitorystorage media embodying software that is operable when executed to, inreal time: capture, by a single sensor, a plurality of images of a user;determine, based on the plurality of images, one or more short-termcardiological signals of the user during a period of time; estimate,based on the cardiological signals, a first short-term emotional stateof the user; determine, based on the plurality of images, one or moreshort-term neurological signals of the user during the period of time;estimate, based on the neurological signals, a second short-termemotional state of the user; compare the first estimated emotional stateto the second estimated emotional state; and in response to adetermination that the first estimated emotional state corresponds tothe second estimated emotional state, determine, based on the firstestimated emotional state and second estimated emotional state, ashort-term emotion of the user during the period of time.
 2. The mediaof claim 1, wherein the software is further operable to: detect a faceof the user in the plurality of images; determine, based onphotoplethysmogram (PPG) data obtained from the plurality of images, oneor more of the short-term cardiological signals of the user; determine,based on facial action coding system (FACS) data obtained from theplurality of images, one or more of the short-term neurological signalsof the user; determine, one or more neuroanatomical signals of the userbased on the plurality of images; and confirm the short-term emotion ofthe user based on a consistency between the neuroanatomical signals andthe short-term emotion.
 3. The media of claim 2, wherein the FACS datais based on detected characteristics of the user's eye brow, lip width,mouth, teeth visibility, chin lines, nasolabial lines, nose lines, orcheek-based impression of smile lines.
 4. The media of claim 2, whereinone or more of the neuroanatomical signals are detected physicalattributes of the user.
 5. The media of claim 1, wherein the software isfurther operable to determine one or more of the cardiological signalsusing a linear model based on a sympathovagal balance (SVB) of the user.6. The media of claim 5, wherein the software is further operable to:determine one or more of the cardiological signals using a non-linearstatistical model based on heartrate dynamics; and determine that theshort-term emotion corresponds to a positive emotion based onconcurrently sensing the SVB and heartrate dynamics and based ondetermining that the SVB and the heartrate dynamics are consistent withpositive emotion.
 7. The media of claim 1 wherein the determination ofthe cardiological and neurological signals are performed concurrently.8. A method comprising, in real time: capturing, by a single sensor, aplurality of images of a user; determining, based on the plurality ofimages, one or more short-term cardiological signals of the user duringa period of time; estimating, based on the cardiological signals, afirst short-term emotional state of the user; determining, based on theplurality of images, one or more short-term neurological signals of theuser during the period of time; estimating, based on the neurologicalsignals, a second short-term emotional state of the user; comparing thefirst estimated emotional state to the second estimated emotional state;and in response to a determination that the first estimated emotionalstate corresponds to the second estimated emotional state, determining,based on the first estimated emotional state and second estimatedemotional state, a short-term emotion of the user during the period oftime.
 9. The method of claim 8, further comprising: detecting a face ofthe user in the plurality of images; determining, based onphotoplethysmogram (PPG) data obtained from the plurality of images, oneor more of the short-term cardiological signals of the user;determining, based on facial action coding system (FACS) data obtainedfrom the plurality of images, one or more of the short-term neurologicalsignals of the user; determining, one or more neuroanatomical signals ofthe user based on the plurality of images; and confirming the short-termemotion of the user based on a consistency between the neuroanatomicalsignals and the short-term emotion.
 10. The method of claim 9, whereinthe FACS data is based on detected characteristics of the user's eyebrow, lip width, mouth, teeth visibility, chin lines, nasolabial lines,nose lines, or cheek-based impression of smile lines.
 11. The method ofclaim 9, wherein one or more of the neuroanatomical signals are detectedphysical attributes of the user.
 12. The method of claim 8, wherein thedetermination of one or more of the cardiological signals comprisesusing a linear model based on a sympathovagal balance (SVB) of the user.13. The method of claim 12, further comprising: determining one or moreof the cardiological signals using a non-linear statistical model basedon heartrate dynamics; and determining that the short-term emotioncorresponds to a positive emotion based on concurrently sensing the SVBand heartrate dynamics and based on determining that the SVB and theheartrate dynamics are consistent with positive emotion.
 14. The methodof claim 8, further comprising concurrently performing the determinationof the cardiological and neurological signals.
 15. A system comprising:one or more processors; and a non-transitory memory coupled to theprocessors comprising instructions executable by the processors, theprocessors operable when executing the instructions to, in real-time:capture, by a single sensor, a plurality of images of a user; determine,based on the plurality of images, one or more short-term cardiologicalsignals of the user during a period of time; estimate, based on thecardiological signals, a first short-term emotional state of the user;determine, based on the plurality of images, one or more short-termneurological signals of the user during the period of time; estimate,based on the neurological signals, a second short-term emotional stateof the user; compare the first estimated emotional state to the secondestimated emotional state; and in response to a determination that thefirst estimated emotional state corresponds to the second estimatedemotional state, determine, based on the first estimated emotional stateand second estimated emotional state, a short-term emotion of the userduring the period of time.
 16. The system of claim 15, wherein theprocessors are further operable to: detect a face of the user in theplurality of images; determine, based on photoplethysmogram (PPG) dataobtained from the plurality of images, one or more of the short-termcardiological signals of the user; determine, based on facial actioncoding system (FACS) data obtained from the plurality of images, one ormore of the short-term neurological signals of the user; determine, oneor more neuroanatomical signals of the user based on the plurality ofimages; and confirm the short-term emotion of the user based on aconsistency between the neuroanatomical signals and the short-termemotion.
 17. The system of claim 16, wherein the FACS data is based ondetected characteristics of the user's eye brow, lip width, mouth, teethvisibility, chin lines, nasolabial lines, nose lines, or cheek-basedimpression of smile lines.
 18. The system of claim 16, wherein one ormore of the neuroanatomical signals are detected physical attributes ofthe user.
 19. The system of claim 15, wherein the processors are furtheroperable to determine one or more of the cardiological signals using alinear model based on a sympathovagal balance (SVB) of the user.
 20. Thesystem of claim 15, wherein the processors further operable to:determine one or more of the cardiological signals using a non-linearstatistical model based on heartrate dynamics; and determine that theshort-term emotion corresponds to a positive emotion based onconcurrently sensing the SVB and heartrate dynamics and based ondetermining that the SVB and the heartrate dynamics are consistent withpositive emotion.